medit malloc problem of RTEMS (cloned)

Original author: chenjin_zhong

Cloned from #4565 (closed):

I find malloc function is called by move_gap function in medit.c. The returned value does not check. At least 32KB of memory is allocated at each time, maybe more than. The returned value "start" should be check to avoid malloc failure. The move_gap function should return immediatelty when malloc failure. the code frament is listed as follows.

static void move_gap(struct editor *ed, int pos, int minsize) { int gapsize = ed->rest - ed->gap; unsigned char *p = text_ptr(ed, pos); if (minsize < 0) minsize = 0; if (minsize <= gapsize) { if (p != ed->rest) { if (p < ed->gap) { memmove(p + gapsize, p, ed->gap - p); } else { memmove(ed->gap, ed->rest, p - ed->rest); } ed->gap = ed->start + pos; ed->rest = ed->gap + gapsize; } } else {
int newsize; unsigned char *start; unsigned char *gap; unsigned char *rest; unsigned char *end;

 if (gapsize + MINEXTEND > minsize) minsize = gapsize + MINEXTEND;
 newsize = (ed->end - ed->start) - gapsize + minsize;
 ** start = (unsigned char *) malloc(newsize);** // TODO check for out of memory
 gap = start + pos;
 rest = gap + minsize;
 end = start + newsize;

 if (p < ed->gap) {
  memcpy(start, ed->start, pos);
  memcpy(rest, p, ed->gap - p);
  memcpy(end - (ed->end - ed->rest), ed->rest, ed->end - ed->rest);
 } else {
  memcpy(start, ed->start, ed->gap - ed->start);
  memcpy(start + (ed->gap - ed->start), ed->rest, p - ed->rest);
  memcpy(rest, p, ed->end - p);
 }

 free(ed->start);
 ed->start = start;
 ed->gap = gap;
 ed->rest = rest;
 ed->end = end;

}

#ifdef DEBUG memset(ed->gap, 0, ed->rest - ed->gap); #endif }