Phase I/II Study to Evaluate the Safety and Tolerability of Avelumab in Combination With Other Anti-Cancer Therapies in Patients With Advanced Malignancies

Inclusion Criteria: * Subjects must be refractory to, or intolerant of, established therapy known to provide clinical benefit for their conditions, or where subjects refuse existing therapies. * Subjects must have measurable disease (RECIST v 1.1) or patients may have bone metastatic disease evaluable by Prostate Cancer Working Group 2 (PCWG2) for subjects with metastatic castration-resistant prostate cancer (CRPC) or according to tumor evaluation criteria best suitable and accepted for the tumor type evaluated. * Eastern Cooperative Oncology Group (ECOG) performance status of 0-1. * Platelets \>= 100 x 10\^9/L (For patients with hepatocellular carcinoma, platelets \>= 70 x 10\^9/L). * Hemoglobin \>= 9 g/dL. * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L. * White blood cell (WBC) \>= 3 x 10\^9/L. * Alanine transaminase (ALT) =\< 2.5 x upper normal limit (ULN) (=\< 5 x ULN for subjects with documented metastatic disease to the liver). * Aspartate aminotransferase (AST) =\< 2.5 x ULN (=\< 5 x ULN for subjects with documented metastatic disease to the liver). * Alkaline phosphatase \< 4 x ULN. * Total bilirubin =\< 1.5 x ULN (In the expansion cohort, subjects with Gilbert's syndrome \[hereditary indirect hyperbilirubinemia\] who must have a total bilirubin of =\< 3 x ULN). * Albumin \>= 3 g/dL. * Serum creatinine =\< 2 x upper limit of normal (ULN) or estimated creatinine clearance \>= 30 ml/min as calculated using the Cockcroft-Gault formula. * Subject has recovered to grade =\< 1 by the National Cancer Institute Common Terminology Criteria for Adverse Events, version 4.03 (NCI-CTCAE v 4.03) from the effects of recent surgery, radiotherapy, chemotherapy, hormonal therapy, or other targeted therapies, with the exception of alopecia. The exceptions for such effects are allowed lab values of =\< grade 2 specified elsewhere in these inclusion criteria. * Life expectancy of at least 12 weeks. * Negative serum pregnancy test in women of childbearing potential within 7 days of first dose of treatment and patients of child-bearing potential must agree to use effective contraception during and after 90 days post dose. A woman of childbearing potential is defined as a premenopausal female capable of becoming pregnant. This includes women on oral, injectable or mechanical contraception; women who are single and women whose male sexual partners have been vasectomized or whose male sexual partners have received or are utilizing mechanical contraceptive devices. * Subjects must have biopsiable disease. For Arms A, B, and C, subjects must have at least two lesions amenable to biopsy and response evaluation. For Arm D subjects should have at least three lesions amenable to biopsy, response evaluation, and radiation. Tumor lesions used for biopsy should not be lesions used as RECIST target lesions. However, if patients in Arm D do not have three separate lesions, patients will be eligible if there are two lesions, in which one is \> 2 centimeters (short axis) and can be used for both biopsy and response evaluation. * Subjects must give informed consent according to the rules and regulations of the individual participating sites. Exclusion Criteria: * Subjects with primary central nervous system (CNS) tumor or CNS tumor involvement. However, subjects with metastatic CNS tumors may participate in this study if the subject is: * \> 4 weeks from prior therapy completion (including radiation and/or surgery) * Clinically stable with respect to the CNS tumor at the time of study entry * Not receiving steroid therapy in treating CNS tumor or CNS tumor involvement * Not receiving anti-convulsive medications (that were started for brain metastases). * Major surgery, radiation therapy or systemic anti-cancer therapy within 4 weeks of study drug administration (6 weeks for mitomycin C or nitrosoureas). Palliative radiotherapy to a limited field is allowed after consultation with the medical monitor at any time during study participation, including during screening, unless it's clearly indicative of disease progression. * Subjects with prior anti-PD-1, anti-PD-L1 treatment. For Arms A and D, subjects may not have had prior 4-1BB treatment. For Arm B, subjects may not have had prior OX40 treatment. For Arm C, subjects may not have had prior 4-1BB or OX40 treatment. * Diagnosis or recurrence of invasive cancer other than the present cancer within 3 years (except basal or squamous cell carcinoma of the skin that has been definitively treated). * Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (\>= New York Heart Association classification class II), or serious cardiac arrhythmia requiring medication. * Active infection requiring systemic therapy. * Treatment with an investigational anti-cancer study drug within 4 weeks prior to study drug administration date. * Concurrent therapy with approved or investigational anticancer therapeutics. * Known prior severe hypersensitivity to investigational product(s) or any component in its formulations, including known severe hypersensitivity reactions to monoclonal antibodies (NCI CTCAE v 4.03 grade \>= 3). * Current use of immunosuppressive medication, EXCEPT for the following: a. intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection); b. systemic corticosteroids at physiologic doses =\< 10 mg/day of prednisone or equivalent; c. steroids as premedication for hypersensitivity reactions (e.g., computed tomography \[CT\] scan premedication). * Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent. Patients with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible. * Prior organ transplantation including allogenic stem-cell transplantation. * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome. * Hepatitis B virus (HBV) or hepatitis C virus (HCV) infection at screening (positive HBV surface antigen or HCV ribonucleic acid (RNA) if anti-HCV antibody screening test positive). * Vaccination (live attenuated virus) within 4 weeks of the first dose of avelumab and while on trials is prohibited except for administration of inactivated vaccines. * Persisting toxicity related to prior therapy (NCI CTCAE v. 4.03 Grade \> 1); however, alopecia, sensory neuropathy grade =\< 2, or other grade =\< 2 not constituting a safety risk based on investigator's judgment are acceptable. * Other severe acute or chronic medical conditions including colitis, inflammatory bowel disease, pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study. * Medical, psychological or social conditions that may interfere with the patient's participation in the study or evaluation of the study results. * Pregnancy or lactation. * Men whose partner is a woman of child-bearing potential, (i.e. biologically able to conceive), and who is not employing two forms of highly effective contraception. Highly effective contraception (e.g. male condom with spermicide, diaphragm with spermicide, intra-uterine device) must be used by both sexes during the study and must be continued for 90 days after the end of study treatment. Women of child-bearing potential is defined as sexually mature women who are not surgically sterile or who have not been naturally postmenopausal for at least 12 consecutive months (e.g., who has had menses any time in the preceding 12 consecutive months). * A diagnosis of active scleroderma, lupus, or other rheumatologic disease which in the opinion of the treating radiation oncologist precludes safe radiation therapy. * Has had prior radiation therapy within the past 3 months where the high dose area of the prior radiation would overlap with the high dose area of the intended radiation based on the judgement of the treatment oncologist.