shasoosh, that is exactly what the code I posted will do: skip the first post on the first page and then show 4 posts per page without skipping any more.
Unfortunately, you did not show enough code after the query_posts to be able to help find the problem. Maybe you should post the whole index.php in the pastebin and post the link here.
Since I can’t see more code, this is a guess. You may be missing the while test after the query. Try using this:
<?php
$paged = (get_query_var('paged')) ? get_query_var('paged') : 1;
$posts_per_page = 4;
$offset = $posts_per_page * ($paged - 1) + 1;
$args = array(
'posts_per_page' => $posts_per_page,
'paged' => $paged,
'offset' => $offset,
);
query_posts($args);
if (have_posts()) { // You may need a colon here istead of a {
while (have_posts()) { // Same here
?>
<div class="post-meta" id="post-<?php the_ID(); ?>">
<div class="image">
<h1><a href="<?php the_permalink(); ?>" rel="bookmark" title="<?php the_title(); ?>"><?php the_title(); ?></a></h1>
</div>