About 30 per cent of government primary schools don't provide the required two hours of planned physical activity each week, according to a new report.
In addition, students at schools that set aside the time don't actually end up engaging in two hours of moderate to vigorous physical activity.
"Students are spending too much time waiting their turn, setting up equipment or travelling to venues and not enough time on moderate to vigorous activity and learning fundamental movement skills," said NSW Auditor-General Peter Achterstraat.
His report found the quality of physical activity instruction varied between schools and teachers, with many students in primary school lacking the skills needed to participate in a full range of physical activities.
"Less than 40 per cent of Year 4 students have mastered fundamental movement skills," Mr Achterstraat said.
His findings follow damning figures about the health of Australia's children, with one in four now classified as obese.
Physical inactivity is putting an enormous financial strain on the nation's health care system, with almost 13,000 deaths each year linked to a sedentary lifestyle.
The report recommended physical activity be integrated into other parts of the curriculum such as maths and english, and that the skill levels of the primary school teachers be increased.
It also said schools should report to parents on their child's physical activity including their aptitude, attitude, skills and level of activity.
"Schools should also report publicly on their physical activity programs and achievements," Mr Achterstraat said.
"Schools are uniquely placed to ensure most NSW children get at least a minimum amount of physical activity."
Mr Achterstraat rejected fears that placing more focus on sport and fitness will damage a child's academic education.
"Higher levels of physical fitness are linked to improved academic performance," he said.
"We can continue as we are and subsidise inactivity to the tune of hundreds of millions of dollars, or we can make a concerted effort to become more active."