I'm not exactly clear on where the techpriests draw the line. I know that they're okay with variations, since every system seems to have its own gun patterns (IE: Calixis-Pattern bolters are different from Obscura-Pattern bolters)

I know they're okay with innovation, because Explorator Research Outposts are one of the basic settlement types of the Imperium (albiet a less common one.) Plus you do occasionally hear about a Magos who invented something cool.

So where exactly are the lines between Hereteks and Techpriests drawn? Is it entirely a philosophical difference, with Hereteks being those who reject the divinity of the machine? Is it based on rank? (So an Archmagos could design a new plasma rifle, but for a lowly enginseer to do so would be hubris?) Is it based on the nature of the changes made?

Into The Storm  includes a modification for plasma weapons that spreads out the plasma and makes them into super-flamers, and the modification is described as being highly illegal and proscribed by the AdMech, such that players will have to seek out a heretek to modify their weapons if they want plasma flamers. So apparently sawn-off rifles are heresy, but I can't see any overarching methodology to what is and isn't legal.

That's because there isn't one.  What is and is not heresy largely depends on who you ask.  The Ecclesiarchy and Mechanicus are both composed of ideologies that vary from sector to sector, and sometimes you even get groups in the same sector that disagree.