Under federal law, it is illegal the minute you have any marijuana or marijuana seeds. Likewise under NJ law, except if authorized by state law, for medical marijuana production.
Under NJ's medical marijuana law, "alternative treatment centers" grow and dispense medical marijuana, and there is no provision for mass-production growers (who don't distribute). Such a center is "an organization approved by the department [of Health and Senior Services] to perform activities necessary to provide registered qualifying patients with usable marijuana and related paraphernalia in accordance with the provisions of this act". The department accepts applications to operate such a center, and "shall seek to ensure the availability of a sufficient number of alternative treatment centers throughout the State, pursuant to need, including at least two each in the northern, central, and southern regions of the State" (they have to find a need for another center in a location, in order for one to get authorized; this is evaluated every 2 years). There are criminal background checks for those involved with a treatment center. According to this article, there are a maximum of 6 centers to be opened, in Montclair, Egg Harbor Township, Woodbridge, Bellmawr, and Cranbury (existing), and Secaucus (under review). The article implies that 6 is the statutory maximum, but that is the statutory minimum, though probably the regulatory maximum (i.e. the state will not authorize any further centers). It also says that the centers must be non-profit, though the law only requires the first two to be non-profit (though it could be a de facto requirement via the regulatory power of the Dept. of Health). Since they don't seem to be eager to expand availability, there isn't any information on what other requirements there are for operating such a center, though the application form is here.
The 2016 annual report gives information on production at the 5 existing centers, which is in the range of 400-800 lbs annually.