I have exactly the same problem; using fedora linux (that uses IP6 as well). I've run wireshark to monitor the DNS queries:

+ test 1: firefox without default ipv6 setting: DNS response takes 4.5 -5 seconds

Code:

6 3.581485 192.168.7.12 192.168.7.1 DNS Standard query A www.google.co.uk 10 8.587615 192.168.7.1 192.168.7.12 DNS Standard query response A 209.85.227.106

+ test 2: firefox with the ipv6 disable set to true: sub 50 milliseconds.
+ test 3: using ssh to connect to a server using it's server name: 4.5 - 5 seconds.

here is my ifconfig:

Code:

eth0 Link encap:Ethernet HWaddr 00:50:BA:62:xx:xx inet addr:192.168.7.12 Bcast:192.168.7.255 Mask:255.255.255.0 inet6 addr: fe80::250:baff:fe62:f9e7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

see that it is running ipv6 by default

I think we can establish that the DNS lookup to my ISP's nameservers is OK (also tested with another router) the Draytek Vigor 2820 has some sort of problem with ipv6 support and DNS lookups... I can't remember how to switch off ipv6 in linux, going to have a nosey around. This is pretty appalling though.