This paper considers an $M/G/1$ queueing model with single vacation and modified $(p, N)$-policy. The modified $(p, N)$-policy means that when the vacation ends and the server returns to the system, if there are less than $N$ customers but at least one customer in the system, the server begins service with probability $p (0 \le p \le 1)$ or stays idle with probability $(1-p)$ until there are $N$ customers in the system and starts its service at once. By the renewal process theory, total probability decomposition technique and Laplace transform tool, we study the transient queue length distribution of the system, and obtain the expressions of the Laplace transform of the transient queue length distribution with respect to time $t$. Then, employing L'Hospital's rule and some algebraic manipulations, the recursive formulas of the steady-state queue length distribution are derived. Meanwhile, the explicit expressions for probability generating function of the steady-state queue length distribution and the expected queue size are presented. Finally, employing the renewal reward theorem, the explicit expression of the long-run expected cost per unit time is also presented. Numerical examples are provided to discuss the optimal control policy $N^*$ for economizing the system cost as well as the optimal two-dimensional control policy $(N^*, T^*)$ when the vacation time is a fixed length $T (T \ge 0)$.