In version "2.5.6-1 64-bits" the textsize of the "x264 Configuration" popup is correct. But in version "2.6.1 64-bits" the textsize is wrong.
I have the following configuration :
OS: Windows 8 64-bits
size LCD screen: 52 inch
resolution: 1920x1080
custom textsize (DPI): 200%, checkbox "Use Windows XP style DPI scaling" enabled
See here screenshots that I have made of the problem :
http://users.numericable.be/lochsoft/avidemux/avidemux.htm (http://users.numericable.be/lochsoft/avidemux/avidemux.htm)