Many people often say you dont need to anneal that often to maintain an acceptable level of accuracy. This may be true depending on the cartridge. Every firing and sizing cycle will work harden the case creating a different material condition in the neck and shoulder junction. Over time you will lose consistency if the material hardness in the particular areas are not maintained. As far as what will happen? Hard to say exactly, but youll probably get fliers here and there and one day get to a point where your case necks wont be able to provide tension onto the projectiles. Consistency is king.
If youre using high quality brass such as lapua or alpha, annealing them will allow them to last a long time (assuming you annealed properly).
Will annealing shrink your groups? Honest answer is probably not, infact it might cause your groups to grow. You will probably need to change your bushing on your neck die to acheive the desired neck tension. You also need to clean the inside of your necks because an oxide layer will form on the inside of the neck. The oxide layer is rough and will cause massive problems. Groups at 100 yards will probably be negligible. Most group shooting isnt properly analyzed anyways, but thats another story for another time (go to precision rifle blog). The gains from annealing will be improved runout and shooting at distance.
Is this worth it? For most, no it makes no sense to anneal. Factory ammo has come a long way and "acceptable" accuracy can be achieved. The time and price of getting gas and tuning your annealer for the specific cartridge and brass type doesnt work out as you will consume a fair amount of time, gas, and ruin brass getting it done right. In most cases, youre better off scrapping the lot and buying new brass.
If you shoot alot, and consistency matter you, and have the cash, get the AMP annealer and be done with it. I use one and its the best annealing solution on the market. I grew up learning using a torch and holding the case with vise grips. You could also find someone who does have one and pay them to do it for you, services like this can be found on other forums.
TLDR
I do da amp ting every time fo make da kine last mo longa.