When you say ADSL you are implying two things;
Low upload capacity and if you reach close to the maximum upload capacity it will indirectly affect your download speed.
ADSL modems usually have very small CPU's that can't handle too many packets per second.
Shouldn't the modem handle the speed of 12mbps. The modem see the use as big download from one user
No, each users opens many connections. The more users the more connections. Since the modems have to keep track of all these connections and it has limited memory and cpu it has a limit to how many it can handle. It depends a bit on the modems though. Some modern modems are much better in this than old ones or the cheapest you can get.
Apart from that, 20-30 800kbps users active (with traffic!) is a bit much for 2 x 12Mb adsl lines.
This for two reasons:
1. 12Mb adsl hardly ever means you get 12Mb. Usually only 80 to 60% of that.
2. More important, and in line to what ´sten´ already wrote, even on plain download, a user produces upload traffic. So, although the download might not even be saturated, for that traffic the upload already might be! And that will have a direct slowing effect on the download....
One other important item not mentioned. Do you do QoS?
If not, it takes only one P2P user to bring a complete adsl line down by consuming all the upload.
So setup a proper QoS system.