Vaccine Therapy and Sargramostim With or Without Docetaxel in Treating Patients With Metastatic Lung Cancer or Metastatic Colorectal Cancer

Inclusion Criteria: * Histologically confirmed lung OR colorectal cancer * Incurable metastatic disease * Currently available standard treatment not likely to offer a survival advantage or result in superior palliation * Evaluable disease by radiograph * Tumor must currently express carcinoembryonic antigen (CEA) by immunohistochemistry OR CEA \>= 10 ng/mL at any point during disease course * No clinically active brain metastases * Must have had first- and second-line treatment OR declined second-line treatment (part I only) * Patients with colon cancer must have had or have been offered treatment with oxaliplatin (part I only) * ECOG 0-1 * Life expectancy of at least 4 months * Absolute neutrophil count \>= 1,500/mm\^3 * WBC \>= 3,000/mm\^3 * Platelet count \>= 100,000/mm\^3 * Bilirubin normal * Meets 1 of the following criteria: * SGOT and SGPT =\< 2.5 times upper limit of normal (ULN) AND alkaline phosphatase normal * SGOT and SGPT =\< normal AND alkaline phosphatase =\<4.0 times ULN * Hepatitis B and C negative by clinical history and physical exam * Creatinine =\< 1.5 mg/dL OR creatinine clearance \>= 60 mL/min * Proteinuria =\< grade 1 * No known or suspected history of impaired cardiac function as evidenced by baseline echocardiogram * Adequate pulmonary function * No history or clinical evidence of immune deficiency or autoimmunity * HIV negative * No history of or concurrent diagnosis of any of the following: * Altered immunodeficiency * Eczema or other eczematoid skin disorders * Acute, chronic, or exfoliative skin condition (e.g., atopic dermatitis, burns, impetigo, varicella zoster, severe acne, or other open rashes or wounds) * No history of allergy or untoward reaction to prior vaccination with vaccinia virus * No history of severe hypersensitivity reaction to docetaxel or other drugs formulated with polysorbate 80 * No history of allergy to eggs or egg products * No frequent vomiting or severe anorexia * No inflammatory bowel disease * No Crohn's disease * No ulcerative colitis * No active diverticulitis * Neuropathy =\< grade 1 (sensory neuropathy) * No uncontrolled seizure disorder * No encephalitis * No multiple sclerosis * Must be maintaining a reasonable state of nutrition (=\< 10 % weight loss in the past month) * Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with individuals with active or a history of eczema or other eczematoid skin disorders * Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with those with unresolved acute, chronic, or exfoliative skin conditions (e.g., atopic dermatitis, burns, impetigo, varicella zoster, severe acne, or other open rashes or wounds) * Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with any of the following individuals: pregnant or nursing women; children =\< 5 years of age; immunodeficient or immunosuppressed individuals (by disease or therapy), including HIV infection * Not pregnant or nursing * Negative pregnancy test * Fertile patients must use effective contraception during and for at least 6 months after study participation * No other concurrent serious medical illness that would preclude study participation * No concurrent biologic therapy * No other concurrent immunotherapy * At least 6 weeks since prior nitrosoureas or mitomycin * Prior docetaxel allowed (part I only) * No prior docetaxel (part II only) * No other concurrent chemotherapy * No concurrent systemic steroids except for the following: * physiologic doses for systemic steroid replacement therapy * local (topical, nasal, or inhaled) steroid use * no concurrent steroid eye drops * premedication prior to and after docetaxel * No concurrent hormonal therapy * No prior radiotherapy to \> 50 % of all nodal groups * More than 21 days since prior major surgery * No prior splenectomy * Recovered from prior therapy * At least 3-4 weeks since prior cytotoxic therapy