Aiming at the positioning and clamping control problem of the push-and-swing camellia fruit picking machine, the dynamic model of the push-and-swing camellia fruit picking manipulator and the dynamic model of the hydraulic servo actuator were established. The state space equation of the control object was deduced based on these two dynamic models. Based on the traditional sliding mode variable structure control (abbreviated as SMVS control in this paper), the RBF neural network fuzzy sliding mode variable structure adaptive controller (abbreviated as NNFSMVS controller in this paper) is designed, which was proved to be stable by Lyapunov's theorem. Then the manipulator control system was simulated with MATLAB/Simulink, and a SMVS controller was used to contrast with it. The simulation results show that the NNFSMVS controller has a faster response speed, and its maximum trajectory tracking error is 0.0026rad/mm smaller than the maximum trajectory tracking error of the SMVS controller, and it can significantly reduce the control system chattering. Finally, after field experiments, the control response speed of the NNFSMVS controller is between 0.8-1s, which can meet the positioning and clamping requirements of the camellia fruit picking machine.